Watson Knowledge Studio provides easy-to-use tools for annotating unstructured domain literatures, and uses those annotations to create a custom machine-learning model that understands the language of the domain. When human annotators label corpus data to create ground truth to train a machine learning model, i.e., a classifier, it is necessary to consistently align their annotations with established annotation polices.
Annotation polices are difficult to enforce across multiple users, i.e., annotators. Conflicting annotations from different annotators must be adjudicated because inaccurate or inconsistent annotations will negatively impact the performance of the machine learning model. This process requires time-consuming discussion among human annotators until all conflicts are resolved and/or annotation policies are modified in terms of the discussion.